{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/VENV36/lib/python3.6/site-packages/pandas/compat/__init__.py:120: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.\n",
      "  warnings.warn(msg)\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<div class=\"jumbotron\">\n",
    "    <h1 class=\"display-1\">自动微分</h1>\n",
    "    <hr class=\"my-4\">\n",
    "    <p>主讲：李岩</p>\n",
    "    <p>管理学院</p>\n",
    "    <p>liyan@cumtb.edu.cn</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 模型优化"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 在深度学习中，“训练”模型并不断更新模型，使它们在看到越来越多的数据时变得越来越好。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 通常情况下，变得更好意味着**最小化**一个*损失函数*（loss function），即一个衡量“模型有多糟糕”这个问题的分数。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 将拟合模型的任务分解为两个关键问题：\n",
    "    - *优化*（optimization）：用模型拟合观测数据的过程\n",
    "    - *泛化*（generalization）：数学原理和实践者的智慧，能够指导生成出有效性超出用于训练的数据集本身的模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 求导是几乎所有深度学习优化算法的关键步骤"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "神经网络的权重更新\n",
    "\n",
    "$$\n",
    "Q(\\mathbf{w})=\\sum_{i=1}^NQ_i(\\mathbf{w})\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\mathbf{w}_{t+1}=\\mathbf{w}_t-\\eta\\sum_{i=1}^d\\nabla_{\\mathbf{w}}Q_i(\\mathbf{w})\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<img src=\"../img/2_preliminaries/saddle_point_evaluation_optimizers.gif\" width=80%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- 深度学习框架通过自动计算导数，即*自动微分*（automatic differentiation）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 根据设计好的模型，系统会构建一个*计算图*（computational graph），跟踪计算是哪些数据通过哪些操作组合起来产生输出"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 自动微分使系统能够随后反向传播梯度\n",
    "    - *反向传播*（backpropagate）意味着跟踪整个计算图，填充关于每个参数的偏导数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 计算机计算微分的方式"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 手动计算微分，再编写代码\n",
    "- 利用Mathematica, Maple等进行符号计算（symbolic differentiation）\n",
    "- 利用有限差分近似（finite difference approximations）的方法进行数值计算\n",
    "- 自动微分"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#### 手动计算"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 理论研究需要解析解"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "> In this article, <mark>a new objective function is defined  and both this function and tis gradient are derived in closed-from for surfaces and volumes.</mark> This method opens a wide range of possibilities..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 进行数学证明与性质分析\n",
    "- 但是，当只需要在优化的时候计算微分，并不需要这些推导过程"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### 符号计算"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 通过变量符号自动推导微分的表达式"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 与手动计算类似，有可能发现优化问题的解析解的性质\n",
    "- 但是，对于计算微分值却是低效的，很容易导致“表达式膨胀”（expression swell）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "例，对于logistic map:\n",
    "$$\n",
    "l_1 = x\\\\\n",
    "l_{n+1}=4l_n(1-l_n)\n",
    "$$\n",
    "计算$l_n$关于$x$的导数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<center><img src=\"../img/2_preliminaries/expressionSwell.png\" width=80%></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<center><img src=\"../img/2_preliminaries/expressionSwell2.png\" width=80%></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### 数值计算"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 对于多元函数$f:\\mathbb{R}^n\\to\\mathbb{R}$，可以近似计算该函数的微分\n",
    "\n",
    "$$\n",
    "\\frac{\\partial f(\\mathbf{x})}{\\partial x_i}\\approx\\frac{f(\\mathbf{x}+h\\mathbf{e}_i)-f(\\mathbf{x})}{h}\n",
    "$$\n",
    "\n",
    "其中，$\\mathbf{e}_i$是第$i$个单位向量，$h$是一个很小的正数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 对于$n$维向量，数值计算的时间复杂度是$O(n)$\n",
    "- $h$的选择对结果影响较大"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### 自动微分"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 一种获取计算某个值的程序，并自动构建计算该值的导数过程的一般方法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 计算图"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 计算图：反映函数计算依赖关系的有向图（Directed Acyclic Graph, DAG）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 节点（node）：代表变量，可以是标量、向量、矩阵、张量等"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 节点也可以是一个操作（operation），即对一个或多个变量的简单函数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 边（edge）：反映了输入到输出的关系"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "例，给出表达式$e=(a+b)*(b+1)$的计算图，其中$a=2,b=1$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "引入两个中间变量：\n",
    "$$\n",
    "c=a+b\\\\\n",
    "d=b+1\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<center><img src=\"../img/2_preliminaries/computationGraph1.png\" width=80%></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "在边上显示导数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<center><img src=\"../img/2_preliminaries/computationGraph2.png\" width=80%></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 可以得到节点之间的间接影响"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 自动微分计算的两种模式 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#### 正向模式"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 从计算图的一个输入节点开始，逐步移动到图的最终输出节点。在每个节点，将指向该节点的导数求和，表明该输入对该节点的总影响"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 跟踪一个输入节点如何影响计算图上的**每个**节点"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "用正向模式计算每个节点关于$b$的导数\n",
    "\n",
    "<center><img src=\"../img/2_preliminaries/computationGraph3.png\" width=80%></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### 反向模式"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 从计算图的最终输出节点开始，逐步移动到图的输入节点。在每个节点，将从该节点发出的导数求和"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 跟踪计算图的每个节点如何影响**一个**最终输出节点"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "用反向模式计算$e$关于每个节点的导数\n",
    "\n",
    "<center><img src=\"../img/2_preliminaries/computationGraph4.png\" width=80%></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "例，给出函数$f(x,y)=log(x*y)$的计算图"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<center><img src=\"../img/2_preliminaries/computationalgraph.png\" width=80%></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 正向计算将每个基本运算的偏导数添加到计算图中\n",
    "- 反向计算求出每个节点的偏导数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<center><img src=\"../img/2_preliminaries/extended_computational_graph.png\" width=80%></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 向量微分"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 向量微分的基础是***Jacobian Matrix***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 给定函数 $\\boldsymbol{f}: R^n\\to R^m$\n",
    "\n",
    "$$\n",
    "\\boldsymbol{f(x)}=[f_1(x_1,\\cdots,x_n),f_2(x_1,\\cdots,x_n),\\cdots,f_m(x_1,\\cdots,x_n)]\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Jacobian矩阵为\n",
    "\n",
    "$$\n",
    "\\frac{\\partial{\\boldsymbol{f}}}{\\partial{\\boldsymbol{x}}}=\\begin{bmatrix}\n",
    "\\frac{\\partial{f_1}}{\\partial{x_1}} & \\cdots & \\frac{\\partial{f_1}}{\\partial{x_n}}\\\\\n",
    "\\vdots & \\ddots & \\vdots\\\\\n",
    "\\frac{\\partial{f_m}}{\\partial{x_1}} & \\cdots & \\frac{\\partial{f_m}}{\\partial{x_n}}\n",
    "\\end{bmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "例，有一个函数$\\boldsymbol{f}(x)=[f_1(x),f_2(x)]$，另一个函数$\\boldsymbol{g(y)}=[g_1(y_1,y_2),g_2(y_1,y_2)]$，二者的复合函数为$\\boldsymbol{g}(x)=\\left[g_1\\left(f_1(x),f_2(x)\\right),g_2\\left(f_1(x),f_2(x)\\right)\\right]$,\n",
    "则$\\boldsymbol{g}$关于$x$的偏导数为？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$$\n",
    "\\frac{\\partial \\boldsymbol{g}}{\\partial x}=\\begin{bmatrix}\n",
    "\\frac{\\partial}{\\partial x}g_1\\left(f_1(x),f_2(x)\\right)\\\\\n",
    "\\frac{\\partial}{\\partial x}g_2\\left(f_2(x),f_2(x)\\right)\n",
    "\\end{bmatrix}=\\begin{bmatrix}\n",
    "\\frac{\\partial g_1}{\\partial f_1}\\frac{\\partial f_1}{\\partial x} +  \\frac{\\partial g_1}{\\partial f_2}\\frac{\\partial f_2}{\\partial x}\\\\\n",
    "\\frac{\\partial g_2}{\\partial f_1}\\frac{\\partial f_1}{\\partial x} + \\frac{\\partial g_2}{\\partial f_2}\\frac{\\partial f_2}{\\partial x}\n",
    "\\end{bmatrix}\n",
    "\\label{eq:jacobianproduct}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 链式求导（式\\eqref{eq:jacobianproduct}）是两个Jacobian矩阵的乘积"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$$\n",
    "\\frac{\\partial \\boldsymbol{g}}{\\partial x}=\\frac{\\partial \\boldsymbol{g}}{\\partial \\boldsymbol{f}}\\frac{\\partial \\boldsymbol{f}}{\\partial x}=\\begin{bmatrix}\n",
    "\\frac{\\partial g_1}{\\partial f_1} & \\frac{\\partial g_1}{\\partial f_2}\\\\\n",
    "\\frac{\\partial g_2}{\\partial f_1} & \\frac{\\partial g_2}{\\partial f_2}\n",
    "\\end{bmatrix} \\begin{bmatrix}\n",
    "\\frac{\\partial f_1}{\\partial x}\\\\\n",
    "\\frac{\\partial f_2}{\\partial x}\n",
    "\\end{bmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 基本方法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### 例1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "对函数$y=2\\mathbf{x}^{\\top}\\mathbf{x}$关于列向量$\\mathbf{x}$求导"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#### 符号计算"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "假设$\\boldsymbol{x}=(x_1,x_2,\\cdots,x_n)^T$，则\n",
    "\n",
    "$$\n",
    "y=2\\boldsymbol{x}^T\\boldsymbol{x}=2\\sum_{i=1}^nx_i^2\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$$\n",
    "\\frac{\\partial y}{\\partial \\boldsymbol{x}}=\\left(\\frac{\\partial y}{\\partial x_1},\\frac{\\partial y}{\\partial x_2},\\cdots,\\frac{\\partial y}{\\partial x_n}\\right)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$$\n",
    "\\frac{\\partial y}{\\partial \\boldsymbol{x}}=\\left(4x_1,4x_2,\\cdots,4x_n\\right)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### torch计算"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "##### 生成向量$\\boldsymbol{x}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-31T02:37:05.865562Z",
     "iopub.status.busy": "2022-07-31T02:37:05.864988Z",
     "iopub.status.idle": "2022-07-31T02:37:06.545457Z",
     "shell.execute_reply": "2022-07-31T02:37:06.544791Z"
    },
    "origin_pos": 2,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'向量x为tensor([0., 1., 2., 3.])'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.arange(4.0)\n",
    "f'向量x为{x}'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 在计算$\\boldsymbol{y}$关于$\\boldsymbol{x}$的梯度之前，需要一个地方来存储梯度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-31T02:37:06.548917Z",
     "iopub.status.busy": "2022-07-31T02:37:06.548534Z",
     "iopub.status.idle": "2022-07-31T02:37:06.552043Z",
     "shell.execute_reply": "2022-07-31T02:37:06.551427Z"
    },
    "origin_pos": 6,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0., 1., 2., 3.], requires_grad=True)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "'x的梯度为None'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.requires_grad_(True) # 开始在这个张量上记录自动微分的操作\n",
    "f'x的梯度为{x.grad}'  # 张量的属性，记录微分结果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 也可以在生成向量的时候指定需要存储自动微分记录\n",
    "\n",
    "```python\n",
    "torch.arange(start=0, end, step=1, requires_grad=False)\n",
    "```\n",
    "- 令`requires_grad=True`即可"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "##### 正向计算$y$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-31T02:37:06.554961Z",
     "iopub.status.busy": "2022-07-31T02:37:06.554622Z",
     "iopub.status.idle": "2022-07-31T02:37:06.560002Z",
     "shell.execute_reply": "2022-07-31T02:37:06.559374Z"
    },
    "origin_pos": 10,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'y的值为28.0'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = 2 * torch.dot(x, x)\n",
    "f'y的值为{y}'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "##### 调用反向传播函数自动计算$\\boldsymbol{y}$关于$\\boldsymbol{x}$每个分量的梯度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-31T02:37:06.562926Z",
     "iopub.status.busy": "2022-07-31T02:37:06.562592Z",
     "iopub.status.idle": "2022-07-31T02:37:06.629473Z",
     "shell.execute_reply": "2022-07-31T02:37:06.628828Z"
    },
    "origin_pos": 14,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'x的梯度为tensor([ 0.,  4.,  8., 12.])'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.backward()\n",
    "f'x的梯度为{x.grad}'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 例2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "计算$\\boldsymbol{x}$的另一个函数$\\boldsymbol{y}=\\sum_{i=1}^nx_i$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#### 梯度置零"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 在默认情况下，PyTorch会累积梯度，因此需要清除之前存储的梯度值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-31T02:37:06.640282Z",
     "iopub.status.busy": "2022-07-31T02:37:06.639960Z",
     "iopub.status.idle": "2022-07-31T02:37:06.645418Z",
     "shell.execute_reply": "2022-07-31T02:37:06.644814Z"
    },
    "origin_pos": 22,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0., 0., 0., 0.])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.grad.zero_()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#### 正向计算$\\boldsymbol{y}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'y的值为6.0'"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = x.sum()\n",
    "f'y的值为{y}'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#### 反向计算梯度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'x的梯度为tensor([1., 1., 1., 1.])'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.backward()\n",
    "f'x的梯度为{x.grad}'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 向量（矩阵）对向量（矩阵）的梯度"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 当`y`不是标量时，向量$\\boldsymbol{y}$关于向量$\\boldsymbol{x}$的导数的最自然解释是一个矩阵"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 对非标量调用`backward`需要传入一个`gradient`参数，该参数可以认为是输出向量关于自身的梯度"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 实际上，`torch`关于向量微分计算的是一个**向量**和**Jacobian矩阵**的乘积"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "假设一个$m$维向量$\\vec{y}$是$n$维向量$\\vec{x}$的函数，即$\\vec{y}=f(\\vec{x})$，则$\\vec{y}$关于$\\vec{x}$的梯度由Jacobian矩阵表达，\n",
    "\n",
    "$$\n",
    "J=\\begin{pmatrix}\n",
    "\\frac{\\partial y_1}{\\partial x_1},\\cdots,\\frac{\\partial y_1}{\\partial x_n}\\\\\n",
    "\\vdots,\\ddots,\\vdots\\\\\n",
    "\\frac{\\partial y_m}{\\partial x_1},\\cdots,\\frac{\\partial y_m}{\\partial x_n}\n",
    "\\end{pmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 给定一个向量$\\vec{v}$，`torch`的自动微分计算的是$J^T\\cdot\\vec{v}$或者$\\vec{v}^T\\cdot J$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 如果$\\vec{v}$恰好是一个关于$\\vec{y}$的标量函数，$l=g(\\vec{y})$，的梯度，即\n",
    "$$\n",
    "\\vec{v}=\\begin{pmatrix}\n",
    "\\frac{\\partial l}{\\partial y_1},\\cdots,\\frac{\\partial l}{\\partial y_m}\n",
    "\\end{pmatrix}^T\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 则，向量和Jacobian矩阵的乘积就是$l$关于向量$\\vec{x}$的梯度\n",
    "$$\n",
    "\\vec{v}^T\\cdot J=\\begin{pmatrix}\n",
    "\\frac{\\partial l}{\\partial y_1},\\cdots,\\frac{\\partial l}{\\partial y_m}\n",
    "\\end{pmatrix}\\begin{pmatrix}\n",
    "\\frac{\\partial y_1}{\\partial x_1},\\cdots,\\frac{\\partial y_m}{\\partial x_n}\\\\\n",
    "\\vdots,\\ddots,\\vdots\\\\\n",
    "\\frac{\\partial y_m}{\\partial x_1},\\cdots,\\frac{\\partial y_m}{\\partial x_n}\n",
    "\\end{pmatrix}=\\begin{pmatrix}\n",
    "\\frac{\\partial l}{\\partial x_1},\\cdots,\\frac{\\partial l}{\\partial x_n}\n",
    "\\end{pmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 向量$\\vec{v}$就是`backward`函数用于计算向量对向量微分时候需要指定的参数`gradient`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "例如，计算$\\boldsymbol{y}=\\boldsymbol{x}*\\boldsymbol{x}$的偏导数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0., 0., 0., 0.])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.grad.zero_() # 向量x的偏导数置零"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-31T02:37:06.648225Z",
     "iopub.status.busy": "2022-07-31T02:37:06.647897Z",
     "iopub.status.idle": "2022-07-31T02:37:06.653894Z",
     "shell.execute_reply": "2022-07-31T02:37:06.653269Z"
    },
    "origin_pos": 26,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'y的值为tensor([0., 1., 4., 9.], grad_fn=<MulBackward0>)'"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = x * x\n",
    "f'y的值为{y}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'关于x的偏导数为tensor([0., 2., 4., 6.])'"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.backward(gradient=torch.ones(len(y)))\n",
    "f'关于x的偏导数为{x.grad}'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 上述方式等同于下述方法\n",
    "\n",
    "```python\n",
    "y.sum().backward()\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0., 0., 0., 0.])"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "tensor([0., 2., 4., 6.])"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.grad.zero_()\n",
    "y.sum().backward()\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 常用向量微分"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "假设$\\vec{x}$为$n$维向量，则\n",
    "\n",
    "$\\forall\\mathbf{A}\\in\\mathbb{R}^{m\\times n}$，有\n",
    "\n",
    "$$\\nabla_{\\vec{x}}\\mathbf{A}\\vec{x}=\\mathbf{A}^T$$\n",
    "\n",
    "$\\forall\\mathbf{A}\\in\\mathbb{R}^{n\\times m}$，有\n",
    "\n",
    "$$\\nabla_{\\vec{x}}\\vec{x}^T\\mathbf{A}=\\mathbf{A}$$\n",
    "\n",
    "$\\forall\\mathbf{A}\\in\\mathbb{R}^{n\\times n}$，有\n",
    "\n",
    "$$\\nabla_{\\vec{x}}\\vec{x}^T\\mathbf{A}\\vec{x}=(\\mathbf{A^T}+\\mathbf{A})\\vec{x}$$\n",
    "\n",
    "$$\\nabla_{\\vec{x}}\\lVert\\vec{x}\\rVert^2=\\nabla_{\\vec{x}}\\vec{x}^T\\vec{x}=2\\vec{x}$$\n",
    "\n",
    "$$\\nabla_{\\mathbf{X}}\\lVert\\mathbf{X}\\rVert^2=2\\mathbf{X}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 分离计算"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 将某些计算移动到记录的计算图之外\n",
    "- 假设`y`作为`x`的函数计算，而`z`则作为`y`和`x`的函数计算。想象一下，我们想计算`z`关于`x`的梯度，但由于某种原因，希望将`y`视为一个常数，并且只考虑到`x`在`y`被计算后发挥的作用"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 使用`detach()`函数实现分离计算"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0., 0., 0., 0.])"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "'y的值为tensor([0., 1., 4., 9.], grad_fn=<MulBackward0>)'"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.grad.zero_()\n",
    "y = x * x\n",
    "f'y的值为{y}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-31T02:37:06.656825Z",
     "iopub.status.busy": "2022-07-31T02:37:06.656481Z",
     "iopub.status.idle": "2022-07-31T02:37:06.662411Z",
     "shell.execute_reply": "2022-07-31T02:37:06.661807Z"
    },
    "origin_pos": 30,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'u的值为tensor([0., 1., 4., 9.])'"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "u = y.detach() # 分离y获得新变量u,该变量与y具有相同的值，但丢弃计算图中如何计算y的任何信息\n",
    "f'u的值为{u}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'z关于x的偏导为tensor([0., 1., 4., 9.])'"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = u * x  # 继续计算z\n",
    "\n",
    "z.sum().backward()  # 获得z关于x的偏导\n",
    "f'z关于x的偏导为{x.grad}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([True, True, True, True])"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.grad == u  # 关于x的偏导是否与u相等？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-31T02:37:06.665549Z",
     "iopub.status.busy": "2022-07-31T02:37:06.665027Z",
     "iopub.status.idle": "2022-07-31T02:37:06.670720Z",
     "shell.execute_reply": "2022-07-31T02:37:06.670092Z"
    },
    "origin_pos": 34,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0., 0., 0., 0.])"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "'y关于x的偏导为tensor([0., 2., 4., 6.])'"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "tensor([True, True, True, True])"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.grad.zero_()\n",
    "y.sum().backward()   # 计算y关于x的偏导\n",
    "f'y关于x的偏导为{x.grad}'\n",
    "x.grad == 2 * x      # 关于x的偏导是否与2x相等？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 控制流的梯度"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 即使构建函数的计算图需要通过Python控制流（例如，条件、循环或任意函数调用），仍然可以计算得到变量的梯度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-31T02:37:06.687918Z",
     "iopub.status.busy": "2022-07-31T02:37:06.687447Z",
     "iopub.status.idle": "2022-07-31T02:37:06.692256Z",
     "shell.execute_reply": "2022-07-31T02:37:06.691639Z"
    },
    "origin_pos": 46,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "def f(a):\n",
    "    b = a * 2\n",
    "    while b.norm() < 1000:\n",
    "        b = b * 2\n",
    "    if b.sum() > 0:\n",
    "        c = b\n",
    "    else:\n",
    "        c = 100 * b\n",
    "    return c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'a为-0.06573323160409927'"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "'d为-107697.328125'"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "'d关于a的偏导为1638400.0'"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = torch.randn(size=(), requires_grad=True)\n",
    "f'a为{a}'\n",
    "d = f(a)\n",
    "f'd为{d}'\n",
    "d.backward()\n",
    "f'd关于a的偏导为{a.grad}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(True)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.grad == d / a   # 由于f是关于a分段线性，因此可以用d/a检验自动微分计算是否正确"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "幻灯片",
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "rise": {
   "autolaunch": false,
   "enable_chalkboard": true,
   "scroll": true
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "vscode": {
   "interpreter": {
    "hash": "34418ffb6f02e522390adb0e13441cc75f901cd11cccb4f6f613643b4b4d2a0b"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
